Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i.e., a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates.

We focus on categorical attributes of language. Examples of such attributes include sentiment, language complexity, tense, voice, honorifics, mood, etc. Our approach draws inspiration from styletransfer methods inthevision andlanguage literature.

We also show that the network width alleviates these issues.

Yiwen Guo leads the project and serves as the corresponding author.

While many techniques for detecting these attacks have been proposed, theyareeasily bypassed when theadversary hasfullknowledge of the detection mechanism and adapts the attack strategy accordingly. In this paper,we adopt anovel perspectiveand regard the omnipresence of adversarial perturbations asastrength rather thanaweakness.

In this paper, we propose a generative multi-column network for image inpainting.

We show that adversarial training is unnecessary and sometimes counter-productive; we instead cast invariant representation learning asasingle information-theoretic objectivethat can bedirectly optimized.

Adversarial training is an effective methodology to train deep neural networks which arerobustagainstadversarial, norm-bounded perturbations. However,the computational cost of adversarial training grows prohibitively as the size of the model and number of input dimensions increase.